Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

ci(benchmarks): run only the last release #203

Merged
merged 1 commit into from
Sep 11, 2023
Merged

ci(benchmarks): run only the last release #203

merged 1 commit into from
Sep 11, 2023

Conversation

P403n1x87
Copy link
Owner

@P403n1x87 P403n1x87 commented Sep 11, 2023 •
edited
Loading

We speed up the benchmarks CI run by only running the latest release and dev. Any issues caught in this way can be analysed locally if needed.

@P403n1x87 P403n1x87 force-pushed the ci/benchmarks-speedup branch 2 times, most recently from be811da to 55e8073 Compare September 11, 2023 13:16
@github-actions
Copy link

github-actions bot commented Sep 11, 2023 •
edited
Loading

Austin Benchmarks

Running Austin benchmarks with Python 3.10.13

Benchmark Summary

Comparison of dev against 3.5.0.

The following scenarios show a statistically significant difference in performance between the two versions.

Sample Rate Saturation Error Rate Sampling Speed
CPU time [sampling interval: 10] 🟡 🟡 🟢 🟡
RSA keygen [sampling interval: 100] 🟡 🟡 🟡 🔴
Full metrics [sampling interval: 1] 🟡 🟡 🟡 🟢

Benchmark Results

Wall time [sampling interval: 1]

Wall time [sampling interval: 1]

Sample Rate Saturation Error Rate Sampling Speed
3.5.0 118000 ± 3000 1 ± 0 1e-05 ± 6e-06 13.2 ± 0.4
dev 117000 ± 3000 1 ± 0 5e-06 ± 4e-06 13.1 ± 0.3
Wall time [sampling interval: 10]

Wall time [sampling interval: 10]

Sample Rate Saturation Error Rate Sampling Speed
3.5.0 116000 ± 3000 0.543 ± 0.002 1e-05 ± 4e-06 13.1 ± 0.3
dev 116000 ± 6000 0.544 ± 0.002 6e-06 ± 3e-06 13.3 ± 0.5
Wall time [sampling interval: 100]

Wall time [sampling interval: 100]

Sample Rate Saturation Error Rate Sampling Speed
3.5.0 10500 ± 400 0.001 ± 0.0005 3e-05 ± 3e-05 14.7 ± 0.7
dev 10400 ± 200 0.0008 ± 0.0001 3e-05 ± 3e-05 14.6 ± 0.7
Wall time [sampling interval: 1000]

Wall time [sampling interval: 1000]

Sample Rate Saturation Error Rate Sampling Speed
3.5.0 1860 ± 30 0.0001 ± 0.0003 0 ± 0 17.9 ± 0.9
dev 1850 ± 20 0 ± 0 2e-05 ± 7e-05 18.2 ± 0.4
CPU time [sampling interval: 1]

CPU time [sampling interval: 1]

Sample Rate Saturation Error Rate Sampling Speed
3.5.0 52000 ± 4000 1 ± 0 1.6e-05 ± 1e-05 22.4 ± 0.8
dev 51000 ± 5000 1 ± 0 2e-05 ± 1e-05 23 ± 2
CPU time [sampling interval: 10]

CPU time [sampling interval: 10]

Sample Rate Saturation Error Rate Sampling Speed
3.5.0 50000 ± 5000 0.9975 ± 0.0002 2.7e-05 ± 1e-05 23 ± 1
dev 51000 ± 5000 0.997 ± 0.001 2e-05 ± 1e-05 23 ± 2
CPU time [sampling interval: 100]

CPU time [sampling interval: 100]

Sample Rate Saturation Error Rate Sampling Speed
3.5.0 5300 ± 100 0.0012 ± 0.0006 8e-05 ± 6e-05 22.3 ± 0.5
dev 5400 ± 100 0.002 ± 0.001 5e-05 ± 8e-05 22.7 ± 0.8
CPU time [sampling interval: 1000]

CPU time [sampling interval: 1000]

Sample Rate Saturation Error Rate Sampling Speed
3.5.0 949 ± 2 0 ± 0 0.0001 ± 0.0003 26.9 ± 0.6
dev 950 ± 3 0.0 ± 0.0001 0 ± 0 26.6 ± 0.5
RSA keygen [sampling interval: 1]

RSA keygen [sampling interval: 1]

Sample Rate Saturation Error Rate Sampling Speed
3.5.0 40000 ± 1000 1 ± 0 0.0003 ± 0.0002 24.2 ± 0.6
dev 39700 ± 900 1 ± 0 0.0004 ± 0.0004 24.4 ± 0.7
RSA keygen [sampling interval: 10]

RSA keygen [sampling interval: 10]

Sample Rate Saturation Error Rate Sampling Speed
3.5.0 39000 ± 1000 0.98 ± 0.03 0.0004 ± 0.0004 24.5 ± 0.7
dev 39900 ± 700 0.98 ± 0.01 0.0002 ± 0.0003 24.3 ± 0.7
RSA keygen [sampling interval: 100]

RSA keygen [sampling interval: 100]

Sample Rate Saturation Error Rate Sampling Speed
3.5.0 6280 ± 30 0.002 ± 0.001 0.0006 ± 0.0006 27 ± 2
dev 6260 ± 60 0.003 ± 0.003 0.0005 ± 0.0007 29 ± 2
RSA keygen [sampling interval: 1000]

RSA keygen [sampling interval: 1000]

Sample Rate Saturation Error Rate Sampling Speed
3.5.0 941 ± 5 0 ± 0 0.001 ± 0.001 31 ± 2
dev 942 ± 2 0 ± 0 0.001 ± 0.003 32 ± 1
Full metrics [sampling interval: 1]

Full metrics [sampling interval: 1]

Sample Rate Saturation Error Rate Sampling Speed
3.5.0 51000 ± 2000 1 ± 0 3e-05 ± 1e-05 30 ± 1
dev 52300 ± 900 1 ± 0 2e-05 ± 1e-05 29.3 ± 0.5
Full metrics [sampling interval: 10]

Full metrics [sampling interval: 10]

Sample Rate Saturation Error Rate Sampling Speed
3.5.0 51000 ± 2000 1 ± 0 2e-05 ± 1e-05 30 ± 2
dev 51000 ± 3000 1 ± 0 1.6e-05 ± 1e-05 30 ± 2
Full metrics [sampling interval: 100]

Full metrics [sampling interval: 100]

Sample Rate Saturation Error Rate Sampling Speed
3.5.0 10000 ± 300 0.002 ± 0.0004 6e-05 ± 5e-05 33.1 ± 1.0
dev 10000 ± 200 0.003 ± 0.003 5e-05 ± 6e-05 32.9 ± 0.7
Full metrics [sampling interval: 1000]

Full metrics [sampling interval: 1000]

Sample Rate Saturation Error Rate Sampling Speed
3.5.0 1850 ± 20 0.0002 ± 0.0004 0.0001 ± 0.0001 37.8 ± 0.9
dev 1860 ± 20 0.0002 ± 0.0002 2e-05 ± 7e-05 37 ± 1
Multiprocess wall time [sampling interval: 1]

Multiprocess wall time [sampling interval: 1]

Sample Rate Saturation Error Rate Sampling Speed
3.5.0 3300 ± 200 1 ± 0 0.00016 ± 3e-05 290 ± 10
dev 3300 ± 200 1 ± 0 0.00014 ± 3e-05 290 ± 20
Multiprocess wall time [sampling interval: 10]

Multiprocess wall time [sampling interval: 10]

Sample Rate Saturation Error Rate Sampling Speed
3.5.0 3350 ± 60 1 ± 0 0.00014 ± 4e-05 286 ± 6
dev 3270 ± 90 1 ± 0 0.00012 ± 4e-05 292 ± 9
Multiprocess wall time [sampling interval: 100]

Multiprocess wall time [sampling interval: 100]

Sample Rate Saturation Error Rate Sampling Speed
3.5.0 3200 ± 100 0.05 ± 0.02 7e-05 ± 4e-05 300 ± 10
dev 3100 ± 400 0.04 ± 0.02 6e-05 ± 4e-05 270 ± 80
Multiprocess wall time [sampling interval: 1000]

Multiprocess wall time [sampling interval: 1000]

Sample Rate Saturation Error Rate Sampling Speed
3.5.0 3100 ± 200 0.008 ± 0.001 2e-05 ± 3e-05 67 ± 6
dev 3000 ± 100 0.007 ± 0.002 1e-05 ± 2e-05 70 ± 10

@P403n1x87
ci(benchmarks): run only the last release
0050447 
We speed up the benchmarks CI run by only running the latest release
and dev. Any issues caught in this way can be analysed locally if
needed.
@codecov
Copy link

codecov bot commented Sep 11, 2023 •
edited
Loading

Codecov Report

Patch coverage has no change and project coverage change: +0.31% 🎉

Comparison is base (6455c21) 68.37% compared to head (0050447) 68.68%.

Additional details and impacted files 
@@            Coverage Diff             @@
##            devel     #203      +/-   ##
==========================================
+ Coverage   68.37%   68.68%   +0.31%     
==========================================
  Files          27       27              
  Lines        2523     2523              
  Branches      775      775              
==========================================
+ Hits         1725     1733       +8     
+ Misses        460      459       -1     
+ Partials      338      331       -7

see 3 files with indirect coverage changes

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

@P403n1x87 P403n1x87 merged commit 0c58b50 into devel Sep 11, 2023
11 checks passed
@P403n1x87 P403n1x87 deleted the ci/benchmarks-speedup branch September 11, 2023 15:06
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers
No reviews
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
1 participant
@P403n1x87